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analysis is not a raliabla method for simultanaous estimation of 
trait and method factors* The poor performanca is due to an inharent 
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trait and method effacts* Covarianca componant analysis, on tha othar 
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and mathod variation in the multitrait-multimathod mat ix and is 
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Abstract 

Several multivariate statiitioal methodolofiai have been propoied to 
ensure objective and quantitativi evaluation of the multitrait-multimethod 
matrix. The paper examines the perfomanae of confirmatory factor anal= 
yais and covariance component modeli. It is shown, both empirically and 
foroallyi that eonflrmatory factor analysii is not a reliable method for si- 
multaiiaoui estimation of trait and method lactors. The poor performance 
is due to g.n inherent rotational indeterminacy common to aU factor ana- 
lytic models of trait and method effects. Oovariance component analysis, 
on the other handi shows a mors parsimonioui parameterizp.tlon of gen= 
eralj trait, and method variation in the multitrait-multimethod matrix 
and is therefore typically unaffected by rotational indeterminaciei. The 
performance with 23 empirical multitrait-multimethod correlation matri- 
ces was also found iatiifactory. 



1 Qualitative foundations of the multitrait-mul- 
tinaethod approach 

Pivotal to the ^gumants in the paper is the notion that method effacti in behav- 
ioral research ^a (a) siiablej (b) undesirablej (c) producti of many ^potential 
influencea at several lavals of abstraction^ (Fiskej 19S2j p. 82), and that (d) 
"wa have only othBr invalid meaBurss againit which to validate our tesfie; we 
have no -criterion' to check them againit" (Campball, 1969, p* 15). The si^a 
of methods effecti in individual maaaurements danno^ be exactly detirminad 
in a platonic ienia, but method dependence can be assessed in a crude senia 

*R.D. Beck, B,W. Flike, K.G, Joreikag j and D, Rindikopf eenfcributed valuable iugfestioni 
and qiieifeisns. 
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whgn the measuremenfcs change with tha assassment methods. Tha queition is 
how trait validity may be aiieiiad without having to know the ^xact nature of 
method disturbsAcei beforehand. 

This Dap er conceives of trn^itB as conitructi relating an unobiervable magni' 
tude to diffarencei among obiarvable uniti of measurement. For iimpliflcation, 
it ia ^sumed that traits and thair mdicafcore are Unt^ly related. 

hi their wall=known paper, Campbell & Fiike (1959) propoead the multitrait- 
multimethod (MTMM) matruc fonnat as a device to study trait validity acroes 
different aiieisment mathodi* The MTMM matrix ihows a croiied measure- 
ment design based on a simple rationale ^ Itaits (i.e,, latent quantitative char= 
acteristici of the rese^ch units) wcm univei^alj equally manifest over a variety of 
situations and dataetable with a variety of methods. Moit importantlyj traits 
ihauld not change juifc because different assessment methods we used* Hence, 
if there ^e m multiple sets of manures of t traits, each utilising a different 
method of asseismant, and if the methods indeed produce equivaJent measnre- 
ments, then the resulting covM'iance matriK ta^es the form 





+ diag(^(i^i),^ 


^(1,2) i 






















+ diaf(^(i^ijj 






[ 











where 

S is the mt x mt covarianca matrix among all measures , 

1 is a m K m matrix of unit entries in all its elements, 

® symbolizes the Kronecker product operator (of. Bock, 1975), 

St is the covariance matrix among trait measures within each method, and 

diag(^(i^i), . . . , 0(m,t)) i§ the diagonal matruc of uncorralated uniqueness 

components of the mt measures. 

The model described by Equation (1) is reaionable only if all measurements 
are made on the same scale* This is yquivalent to the psych ©metric concept of 
r-equivalent measurement (Lord U Novick, 1968). In the behavioral and social 
sciences, where veiy often diverse methods like test scores, behavioral obser- 
vations, and one-item ratings ^e compared, such strong scale asiumptions are 
usually not warranted. Scale information is typically reg^dsd as arbitrary or 
of Uttle interest and, since Spearman's days, the soci^ sciences have had a tra- 
dition of analyiing eomiatton matrices, effectively neglectinf information due 
to the original scale of measurement* For these reasojii, some terminology de= 
signed to describe valid meaiurement in more general eerms based on correlation 
patterns is preferable to the strict formulation of Equation (1), 



3 



Campbell ic Fkka proposed leveral qualitafciva criteria to judge convergent 
and discriminant validity. These criteria ^e quite popular and appear to be rig^ 
orousj but can be shown to be not quite adequate in borderline cases (Wothke, 
1984) and, because of their complete lack of any statistical basis , need to be re= 
placed by quantitative rules (see, e.g.i Althauserp 1974; Althauser k Heberleinj 
1970; Althauser, Heberlein k Scott, 1971). 

Confirmatory factor analyna (Joreskogj 1966, 1977) and covariance compo- 
mnt analysis (Bockj 1960; Bock U Bargmann, 1966; WOey, Schmidt & Bramblej 
1973) are two quantitative approaches with potential application to multitrait- 
multimethod analysis. Both models are reallocations of the multiv^iate linear 
model and are embedded in an abundance of statistical theory* Otherwisej they 
are structurally distinct and derive from different statistical traditions^ Confir= 
matory factor analysis is rooted in the psychometric tradition of validity theory, 
as outlined by Lord k Novick (1968); covariance component analysis is a mul- 
tivariate generalization of rMidom effects analysis of variance, based on R.A* 
Fisher^s work. 

2 Conflrmatory Factor Analysis of the Multi- 
trait-Multimethod Matrix 

Confirmatory factor analysis (CFA) of the MTMM matrix was first proposed 
by J&eskog (1966^ 1977). Employing essentially the same maximum-likelihood 
estimation techniques as Lawley's (1940) exploratoiy factor analysisi confirma- 
tory factor an^ysis m commonly characterised by addition equality restrictions 
imposed on estimated factor loadings, factor variances and covariancea, and '^n 
unique components of the measured variables. A computer program for CPA 
is available in LISRBL-6 (Joreskog & iorbom, 1986). Assuming multivariate 
normality of factor space and measurement eirorSj maximum-likelihood tests 
among nested models cbjl be performed. In cases of non-normality, the more 
recent work by Browne (1984b) seems promising^ using a weighted least-squares 
estimation approach. 

Factor analysis decomposes the n x p data matrbc X of p of measures on n 
units into an n x ^ matruc 3 of a lesser number h of latent factors: 

X=iA' + E, (2) 

where A is the px matrix of p^tial regression coefficients of observed measures 
regressed onto the latent factor and E is the matrix of unique, uncorrelated 
components. Expressing the population covariance matrices of X as S^j of 3 as 
and of E as ©, respectivelyj Equation 2 implies the cov^iance representation 

^ A*A -f- 0, (3) 

Over the years ^ different types of confirmatory factor models have been pro- 
posed for the multitrait-multimethod matrix. They cbjl be chwacteriaed by 
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how majiy iafcent factori are modeled; whathar these faefcoi^ M-a thought to de= 
icriba trait varianca, method va^i^ca, or both; md whethar the coiralation 
itructura among thise mathods m fraa or restrictad. OccajionaUy, modali with 
coiTalated uniquanesi coafflcienti ware ako appHad (m.g., Staey at aL, 1985), but 
the present paper retains the classic factor analytic notion that uniqua compo- 
nents ara uncoirelatad. With this one restriction^ aU CPA modals of the MTMM 
matrix may be described by pMrticulM- raitriction pattems impoied on tha A 
and # matrices i 



2.1 Itait-only factor analysii 

The iimplast factor analytic modali require that all common variation wnong 
measuras ii due to the latent trait factori and that no covariation is due to 
tha ^iessmant methodi. Different traits may ba coirelatad. Such a trait- 
only model, which ipeciflas that each measure assaiies exactly one trait factor 
(Joraskogj 1971, 1978; Bchmitt^ 1978; Werts, Joraikofi k Lmn, 1972; Warts k 
Linn, 1970) shows the properties of cenganan'c maaiuremant. Whan maarnres 
ara ordered by traits within maaguraii the factor bading matrix for a 3 x 3 
MTMM design takes the form 



0 
0 

0 
0 

0 



0 

0 
0 

0 

0 

0 



0 ^ 

0 

0 

0 

0 
0 

A§,r3 J 



(4) 



and the matrix of factor intarcoirelations is obtained by 




(6) 



The factor struct ura is non -overlapping and oblique. All gero entries in Ay 
and all diagonal entries in #r ^€ fixed (pradateimined) parametari, the nine 
symbolic parameters A^-.^^, and the three symbolic paramaters #r are esti- 
mated from the data. Also estimated are tha nine uniquenass coefficients in the 
diagonal of 0. 

The Campbcll-Fiike critariaj appHad to the trait-only model, appear as in- 
equality constraints on factor loadings and factor intarcowelations. Two of the 
criteriaj ^e rather useful, yet, because Campbell &c Frnkm tacitly ajsumed ho- 
mogeneous reliabilities, tha other two criteria imply a somewhat unintelligible 
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trad^^off bativ^een fche boundMy conditions for ; k ^ Itogi and factor corre- 
latiosis (Wothke, 1984). 

The most intareitinf question for the appited ref^i^cher la how well the con- 
generic trait-only model can be uiid to deicribe and analyze empirical MTMM 
matrices. Critical indicator! for model performance ate 

Idantifldation— uniqnaneii of the model par^neter estimatei. The parameters 
are not identified when different sets of values for one or more esfcimatei 
result in the iame model covariancB matrm fi. A trivial case of non^ 
identification occuri when the model has more paramet^s than elements 
in the model covajiance matrix. Further issues concerning the identiflca^ 
tion problem with MTMM factor analyiii are discussed in Wothke (1984). 

Convergence— numerical evaluation of the parameter estimates* Maximum^ 
likonhood parameter estimates for the covariance structure models consid- 
ered here do not have a cloied-foim iDlution* Eitimatei must be obtained 
iterafcively, using, for example, the Fletcher-Powell or Newton=Eaphion al- 
foritliiiis implemented in the tISREL program. Both numerical methods 
are generally efficient, but cm fail to converge to a final solution, either 
because the number of iterations will exceed the prasent program limit of 
250 or becauife the procesi wUl actually diverge. Common reasons for non- 
convergence are (a) stating values chosen too far from the final solution, 
(b) flat maxima or ridges in the likelihood iurface, and (c) singularity of 
the information matrix in the vicinity of the solution. The latter two cases 
indicate poor model properties* 

Admiisibility— the Fkherian estimation methods employed by LISREL and 
related programi may produce pw^ameter estimates that ia*e not com- 
patible with the measurement model in Equation 2. For instance, nega- 
tive uniqueness components &{ or factor coirelations in excess of 1.0 are 
not uncommon* In formal terms, all eitimafced covariance matrices (herei 
#, ©) are conf:eived as Gramian and must be non=nagative definite. Vi- 
olations would imply a complex-valued meajurement space. Substantive 
consid^ations may produce even stricter criteria whenj for instance, the 
communality of a measure exceeds its known reliability. For all practical 
purpoiei, the emergence of non-admissible parameter estimates indicates 
poor specification of the structural model. 

iModel fit— fit ii evaluated in terms of deviations between the sample and the 
eitimated model covwance matrices. Several measures are conceivable 
and have been proposed in various papers. A popular fit itatistic with 
powerful l^ge-sample chwacteristics is the maximum-likelihood G^^ com- 
puted as 

« iV^ [in jS| - In |S| + trace(Sfi^^) - p] . (6) 
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Under mulfcinonnalifcyj and when fcha correct ifcrucfciiral model is selected, 
is ^ympfcotic^y ;f ^-distributed with 

where p m the order of the covarianee matrix and t m the n!imber of 
independently eitimated model paramgters. When a more restricted or 
modified model is appUedj will follow a non-central x^'distribution* 
Model fit CM only be properly aisesied when the eitimation has converged 
to a?A admisiible soliition. GeneraUyj inadmisiible parameter eitmiates are 
asiodatad with over-flt of the model so that the ^^-statistie wHl be neg- 
atively biased. It is alao well-known that non-normality and non-random 
samplmg wHl bias the G^-statistie in the positive direction. 

Uiing these four criteria (identificationj convergencej admiiiibility, and fit)j 
perfonnance of conganeric trait-only factor analyik was evaluated with 23 em- 
pirical multitrait-multimethod matrices* TM datasets were obtained from publi« 
cationi in psychological, sociological, educational and marketing research jour- 
nals comprising a probably typical coUection of MTMM matrices from the^e 
flalds. The sample of the datasets was biased^ many MTMM matrices with very 
imall sample aizej incomplete measurement design, and/or correlations based on 
pairwiie deletion or other non-Grajnian procedures were rejected. Converselyj 
datasets that h^d been reanalyzed in the literature had a higher chance to be 
included in the sample. Origin and nature of the matrices are described in the 
Appendix* 

Results of the an^ysis are summmzed in Table 1. No glob^y under= 
identified solutions of the congeneric tr^t-only model were observed— all the 
information matrices were of full rank* Convefgence problems ©ccuired with 
the three datasets ^Attitudes to Authority (Burwen Is Campbell)^, **Personal- 
ity Itaits (Kelley & Krey)^, and ^Job Behavior (Dickinson & Tice)^* Inspection 
of intennediate solutions for these datasets suggested local under-identification 
m the likely reason for non-converg en *e* Li addition to three non-converged so= 
lutionsi the model produced inadmissible parameter estimates for seven further 
datasets* This left 13 of the 23 datasets with admissible congeneric solutions* 

The <j^-statistic shows acceptable fit for just two of the remaining 13 dati^eti 
[""Smoking and Capital Punishment (Jaccajd)^ ^d "'Three Attitudes (Flamer, 
Sample l)"]| Jacc^d's dataset should not be given much weight, howeverj con- 
sidering the small N of 35* 

Table 2 shows the parameter estimates for the Flamer (Sample 1) dataset. 
The traits ^e "Attitude towards Discipline in Children^ (ADC), "Attitude to- 
wards Mathematics^ (AM), and ""Attitude tow^ds the Law^ (AL)* All assess- 
ment methods are paper-and-pencilj but comprise different item types and re- 
sponse formats: dichotomous Likert sc^es, Thursfcone scales, and the semantic 
differential (SD) technique. Since the maximum-likelihood G^-statistic indicates 




Table 1: Congeneric trait^only analyiii 
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Tabla 1^ — Continued 
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Table 2t Congenaric trait-only esfcimatei of the Flamer (eample 1) data 
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m aceeptabla model fltj aofivsrgtnhdidit^ m confirmed for all niiKa fcralt-mafchod 
eonibinationi. One can Hbq iee that tJia relative precision of xttaaasurement m 
coniiifcantly higher for meaautii of "Attitude towards Mathem^atlci" than for 
**Atfcitnde towardi the Law*. Some eaution m indicated for the terait ^Attltuda 
towardi Diaciplina in Childten^ wMch ghowi a larger vmatioa i^m the size ©f its 
factor loadingi. The heterogenaily of the factor loadingi ii mar-^ginally lignifi- 
canti tasted againit a modal lyi^hiquj factor loading for each toraitj there is a 
fit increaae of Diff-G^ s lS.l{dJm^J^ m O.020) for the congaaeir-^c modeL 

Ducriminant validity be twain the trait eoncipti can be ju^^ged from the 
estimated factor coiralatioii mM^r* Evidently **Atfcitiide t^^wards Mathe- 
matics^ is virtually unrelated totlii othM two traitSj while the disatfcanuated 
correlation coefficient of $fs,Ti ^Mihoyrm a mild ^iociation beteween the atti» 
tudes towards the law and toWafJidlieipline, 

M summary: Eitim at ion ©f the eongenerlc trait-only factor ^nodel wiU of- 
ten converge to m admiiiibl% iolution, but the fit to empirics al multitrait= 
multimethod matricei tends to be poor. These problomi of iiio--dal misfit can 
be blamed on the data— moit ftmplrieal dataseti did not supporE the notioni ©f 
convergent and discriminant v^jdity, In fete two eases where the r^nodel showed 
a good fit, the assessment metjiodi appealed to differ only in relatively minor 
aapacts of queition wording* 



2.2 Trait-method factor analysis 

As a less reitrictive ^temafcive to the trmit-only modelj sever d authors have 
suggested to include additional mathod factors (Althauier, 1974^ Althauser <k 
Heberlein, 1970| Althaaser, He^barliliij k Scott, 1971; Joraikof, 1^371; Kalleberg 
U Kluegel, 19T5| Schmitt, 1978; Werti, JSxeskogj k Linn, 1972; ^V^erts U Linn, 
1970; Werts, Linn k Joreskogj Wl], The xationde for added meteliod factors is 
that, apart from expresiing trait variation ^ measurei may also be -*^owelated be- 
cause they share the same ^leisniint metHod. Method factori ^ ^Tll purportedly 
account for systematic variatioii due to th«e shMed method cont^^onents. 

For a 3-tralt-by-3-method iniaiiirement design, the factor lc»ading matrix 
Ar,A* of the trait-method faefcof inadel is simply conitructed by a^ngmenting 
from Equation 4 with three additional method factors ^ij/i2,ani /^a: 

\^ 
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(8) 




This expanded model laems to be at least parfciaUy mofcivatdb^ hopa^ of 
finding a itatiitical procruitee method able to elimmata method iffacte from 
l^ha measurements. The argumant goes as followi^ 

If trait factors ^e unco^elated with method facfcorij the feipaefcive 
facfcor scores should also be uneoiTalated in the populstion. The 
trait scores would be retained for further analysis of ^mathoWree^ 
trait measureSj while method scores would be rejected m **traiMgii^ 
measurement ^fcifaets. la this sense the fcrait-method factor model 
can possibly be uied to separate trait and method gompoiienti, 

^Bvan though this proposition sounds somawhat fantasfcicj the conarmatory trait- 
tenafehod factor modal can oasily bo roitricted to independence betwiin traiti and 
^^lethods specif yinf the factor intercoiTalatioa matrix as 
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"^- ha remainder of this section wiU examine model perforniaiic^ ^illi a block- 
fcitiagonal type of GOn*alatioii itructure described in Equatioii 9. Wlien clarity is 
r-i-^quired, explicit reference is made to the trait'inethod indBpend^mmodelB. 

The difference between trait-method and trait-only factor nioialiii not just 
a-^ matter of quantity of factors. The trait-only model, conc^lv^d in the psy- 
c^Zhometric tradition of parallel measurement of a single latent ti-ail, daieribes 
n_=.on-overlapping factor concepts. The trait-method factor modalj on the other 
l^ajidj expresses the systematic variance of each measure as th^ Uiieareoinbina- 
t^aon of two latent factors and is ovsriapptng. In coniequencgj tha trait-method 
i^iodel does not reflect the paraUolity concepti of classical tmnt thoty and is 
l^Mgely iiTelevant to the asiassment of factorial v^idities in a lat ofmaasi\res. 
^^ha trait-method model rather describes a metric linear decoiTijjoiltloii of an 
oS^sarved measurement structure dkt^tly related to deGompositioninodeis ap- 
psjoaches in the tradition of Baals at ah (1968). Furtherinore, the moiil appears 
deviate substantially from the Campbell & Piska (1959) trait cOMplion— the 
v^aUdity criteria of the original paper fail to establish re^onabla boundaries of 
th=Lm parameter space (Althauser, 19?4| Althauser & Heberlain, l^rOiAlthauserj 
ICaberlein, U Scott, 1971). 

Performance of the trait-method factor model is iummari^ed in Table 3. 
T^hm model was globally unidentified with four datasets [^Ihfeellitinca and Ef- 
t^rt (Mayo)^, ^'Intelligence and Alertneis (Thorndike)'* , ^Authorfy (Burwen 
= CampbeU)^, and ^Drives in Rats ( Anderson)**], The coff elatio/i maHces of 
ti^'^s^ datasets were too imall, containing fewer empirical coJTilafciojicaefflcienfci 
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Table 3; Itaifc-method independence factor analyeii 



Dataaet 


Admissible 

Identified Converged Solution df 


N 


Intelligence 
and Effort 
(Mayo) 


no 




Intelligence 
Alertness 
(Thorndilce) 


no 




Popularity & 

Expansiveness 

(Borgatta) 


no no 3*7 5 


125 


Smoking 
and OP 
(Jaceard) 


no: the 23rd (of 31) parameter may not be identified (^^^ 




Leadership 
Study (Sum^ 
mars, et aL) 


no: the 2iit (of 31) parameter may not be identified (^^^ 




Authority 
(Burwen Ss, 
Oampbell) 


no 




Drives in 
Rati 

(Anderson) 


no 




Involvement 
Components 
(Arora) 


diverged no 403,9 3,2 


96 


Job Behav- 
ior (Dickinson 
<k Tiee) 


diverged 

no- the 3 lit p^ameter (of 33) may not be identified {67) 




Three Atti- 
tudes (Flamer, 
Sample 1) 


117+ 

no: the 6fch p^ameter (of 33) may not be identified (Ag^^^ 


) 


Three Atti- 
tudeSj Flamerj 
Sample 2) 


145+ 

no: the 6th parameter (of 33) may not be identified (Ag^^^ 


) 


Streas 
Measures 
(Kajit & Most) 


diverged no 3892.2 12 


80 


13., 



Table 3— Continued 



Admiiiible 

Dafeastt Identified Converged Solution df N 

Job Perform- _ 

anca diverged no 5580.6 12 113 

(La^vler) 



Moral ^ '' 

Dilamma diverged no 216SS.9 12 487 

(Shepherd) 



Contracep= 






tivfi (Kot'^ 
handapani) 




no 534 33 100 


Attitudes to 






the Church 




no 21.7 33 189 


(Ostroni) 






Drug Uie 






Reports 




no 93,5 33 190 


(Stacy et al.) 






Clinical 




diverged 


Clerkships 
(Boodoo) 


noi the 


2nd (of 58) parameter may not be identified (Ai^^^ ) 


Personality 






*^aiti 

(Kelley Krey) 


no: the 


23rd (of 31) paramafcer may not be identified (^^3,^1) 


Desirability 






(Jackson 




no no 410J 134 480 


Singer) 






Interaction 






Process Vari. 




no no 100.5 62 125 


(Borgatta) 






Guilford- 






Martin Fact. 
(Carroll) 




n© 112.5 62 110 



Assessment 



(Kelly ^ no 57.5 62 124 

Fiike) 




than there wera parameters to be estimated. Seven further cases also resulted 
in unidentiSed solutions even though the number of correlation coefficients ex- 
ceeded the number of independent model parajneters [^Smoking and Capital 
Puniahment (Jaccwd)^j "Leadership Study (Summers et al.)^, "Job Behavior 
(Dickinson I. Tice)^j *'Three Attitudes (Flamarj Sample 1)^/ "Three Attitudes 
(Flamerj Sample 2)", ^CHnical Clerkshipa (Boodoo)^j and "Personality Ttaits 
(Kalley Is Krey)-]. The paramefcers involved vary even among similar struc= 
fcured dataseta so that one might expect empirical underidentification, Yetj the 
frequency of unidentified solutions appeaxs luipiciouily high. 

Among the remaining twelve datasetSj solutions converged in five cases. Gon= 
vergenca failed in seven cases. Yet, in no casej converged or notj was the solution 
admissible. Table 4 ihows the inadmissible solution for the Assessment (Kt^Jley 
U Fiike) data aj a typical example. Traditional interpret at ion of this well-kno%vn 
dataaet has occaaionally concluded that the Staff ajid Self Rating method fac- 
tors should be combined because their coiralation is so excessive (Joreskogj 1971; 
Brownej 1984a). ^nch a decision supposes that the estimates are inadmissible 
a consequence of problematic sample conrelation matrix rather than because 
of a structural deficiency of the model- This does not seem to be the case. First, 
if inadmissibility was due to sample problems , one should also be able to find 
datas^ts that have an admissible solution. The seaxch for such a dataset was 
negative, as documented in T^ble 3. Second, a strong point can be made that 
the trait^method model is conceptually flawed. Suppose that aJl measures in a 
particular study share some common vajiance due to any kind of shared circum= 
stances. Ooniinon variance can be shared for a number of reasons, for instance 
(1) choice of simile measurement situations, (2) choice of similar traits, or (3) a 
strong general factor of individual differences. The three interpretations relate 
the common variance to method, trait, or neutr^ concepts, respectively, but 
they cannot possibly be distinguished on empirical grounds in a single MTMM 
study^ 

This conceptual identification problem has a direct numerical equivalent. 
The common variance may be accounted for by either the covariance structure 
due to trait factor^ or by the method factor structure. Existence of the indeter= 
minacy can aasHy be demonstrated for two more restricted forms of trait-method 
factor analysis. 

Two^factor modeh Suppose the factor coirelation matrix is restricted so (a) 
that all traits ^e perfectly correlated with each other and (b) all methods 
are likewise correlated with unity among themselves. Such a model is 
equivalent to an exploratory factor analytic solution with two orthogonal 
factors and the loading matrix 
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Table 4: IVaxt-mathod independence factor analysist of the Kelly & Fiske aa- 
sessmant data 



Factor loading matrix A^^i 
























TrMt factor! 




Meth 


□d factors 


Tin Inii^TiR^Q 


Method Trait 


A 


0 


s 


p 


I 


Staff 




Self 


Estimates & 


A 


.86 


.00 


.00 


.00 


.00 


-.07 


,00 


.00 


.26 


Staff 0 


.00 


,83 


.00 


.00 


.00 


-.05 


.00 


.00 


.31 


Ratings S 


,00 


.00 


.60 


.00 


.00 


.09 


,00 


.00 


,62 


P 


.00 


.00 


.00 


.89 


.00 


.14 


.00 


.00 


,20 


I 


.00 


.00 


,00 


.00 


.72 


.15 


.00 


.00 


.45 


A 


,84 


.00 


.00 


.00 


.00 


.00 


.13 


,00 


.29 


Teaniniate 0 


.00 


.83 


.00 


.00 


.00 


.00 


.28 


.00 


.47 


Ratingi 8 


.00 


.00 


.68 


,00 


.00 


.00 


,35 


.00 


.41 


P 


.00 


.00 


.00 


,18 


.00 


,00 


.58 


.00 


.65 


I 


.00 


.00 


.00 


.00 


.57 


.00 


.50 


.00 


.43 


A 


.56 


.00 


.00 


.00 


.00 


.00 


.00 


.16 


,68 


Self C 


.00 


.45 


.00 


.00 


.00 


,00 


.00 


.24 


.76 


Ratings 8 


.00 


,00 


.44 


.00 


.00 


.00 


.00 


.28 


.74 


P 


.00 


.00 


.00 


.43 


.00 


,00 


.00 


.41 


.66 


I 


.00 


.00 


.00 


.00 


.67 


.00 


.00 


.57 


.28 


Factor correlations 
























A 


C 


S 


P 


I 


staff 


Mate 


Self 




A 


1.00 




















C 


.56 


1.00 


















s 


-.39 


-.43 


1.00 
















p 


.33 


.62 


-.07 


1.00 














I 


,54 


.30 


=.03 


.46 


1.00 












Staff 


.00 


.00 


.00 


.00 


.00 


1.00 








Mate 


.00 


.00 


.00 


.00 


.00 


.88 




1.00 






Self 

■ — ^j-i , - ~ ' 


.00 


.00 


.00 


.00 


.00 


-2.34 




-.01 


1.0 





= 57.64 ~ ^ ^ p ^ 0.503 

df ^ 62 AT s 105 



It is weU known fchafc only 2p - 1 of tht 2p paramatars in matrix 10 can 
be eitimatedj ona parametar has to ba iat to zero in order to fix the 
orientation of the factors (Anderion h Rubinj 19S6| Dunn, 1973; Joreskog 
k SDrbom, 1979, pp. 4C^4S). The trait-method factor model does not 
incorporata such a constraint atid is tharafora uridantified* 

Equally weighted indicators? Suppoaa the factor model is simpliflad so that 
all astimatad loadinfi of a' given factor hava the sajna valua. Such a 
solution, prasanted in equation 11^ specifies that v^iance of a given factor 



interpretation of tha 


factors. 




f 


0 


0 


A4 


0 


0 




0 


Ag 


0 


A4 


0 


0 




0 


0 


A3 


A4 


0 


0 




Ai 


0 


0 


0 


As 


0 




0 


A^ 


0 


0 


As 


0 




0 


0 


As 


0 


As 


0 




Ai 


0 


0 


0 


0 


Ae 




0 


A2 


0 


0 


0 


Ag 




I 0 


0 


As 


0 


0 


Aa 


J 



Xrfj, can be expreased as the product of a design matrix A^^ and a diagonal 
matrix D^^ of factor loadings 







i 


0 


0 


1 


0 


0 ^ 






0 


1 


0 


1 


0 


0 






0 


0 


1 


1 


0 


0 






1 


0 


0 


0 


1 


0 






0 


1 


0 


0 


1 


0 






0 


0 


1 


0 


1 


0 






1 


0 


0 


0 


0 


1 






0 


1 


0 


0 


0 


1 






0 


0 


1 


0 


0 


1 J 

















The modal equation then becomes 



/ Ai 0 
0 X2 
0 0 



0 
0 

V 0 



0 
0 
0 



0 0 
0 0 

As 0 



0 
0 
0 



0 
0 
0 



A4 

0 

0 



Da 



0 

As 
0 



0 
0 
0 



0 
0 

Ae J 



(12) 



(13) 



Since A has rank * + m — 1 ^ only at most t-hm—1 functions of the t^m 
factors m aitimable (Graybill, 1961, pp. 228-229). Wothke (1984) has 
shown that any singla paramatar in Equation 13 can ba fixed in a way 
that iolves the identification problem. Such a solution wouldj however, be 
wbitrary and render the remaining pwameter estimatei meaningless. 
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DiicUision of idenfcificafcion condifcions for the general form of the trait- 
method model is still difficult and far from conclusive. It can, for instance^ be 
shown that the trait-method independence model doei not fulfill the iufficiency 
conditioni for factor identificatbn outlined by Anderson ^ Rubin (1956), Jen- 
nrich (1978), and Joreikog & Sorbom (1979). Accordmg two these sufficiency 
conditions, wa may have a case of rotatiDnal underidentifieation on our hands, 
but there is no conclusive proof. For the moment wr shaU be iatisfied that the 
caae of equally weighi&d indiaaiors, a textbook example of a simple structure 
decomposition, cannot be identified. The relative orientation of the trait and 
method subspaces remains undefined. 

2,3 Discussion 

Twenty three empirical MTMM matrices were analysed with trait-only and trait- 
method independence factor models. Neither model showed perfect performance 
with all datasets. The trait-only model converged to admissible and identified 
solutions in more than half the cases, but model fit was acceptable only in two 
cases. Trait-only factor analysis is the most desirable model, but most empirical 
coirelation matrices do not conform. 

On the other hand, analyses with the trait-method indBpendence model failed 
completely. The practical consequence of these results is that the trait =mat hod 
model is not applicable to any of the 23 datasets. Apparently, the factor analytic 
treatment of the multitrait-multimethod matrix has reached its limits with the 
trait-method model already. The seemingly sensible approach of reducing the 
syitematic variance into sets of trait Md method factors cannot be applied. The 
solutions are either not identified or ^e not admissible. Either case precludes 
substajitive interpretation of the parameter estimates. The reason is that the 
structural conception of the measurement design is deficient. The trait-method 
model appears to be overparMietarised with the consequence that the solutions 
are rotationally undetermined. 

3 Covariance Component Analysis 

Covariance component analysis (CCA) was first introduced by Bock (1960) and 
Bock & Barfmann (1966) as a miiltivariate random model for factorial measure- 
ment designs. The method was originally designated as ^covmance structure 
ajialysia^, the term is avoided here because it has since become synonymous with 
the more general class of structural equation models. A successful application by 
Bock, Dicken, & Van Pelt (1969) inveitif ates the effects of content-acquiescence 
interaction in MMPI scales. 

Covm^ce component analysis explicitly accounts for the general level of 
cov^iation common to aU meaanres in the design, trait variation, and plus 
method variation, but contains only those* poxameters that, at least in princi- 
ple, can be estimated, CCA thui avoids the indeterminacy encountered with 



trait-method factor analysis. Apparent problemi with CCA were rooted in tra- 
ditionally itrict ^iumptioni about scale and e^or vaj-iance of each measure, 
untenable for MTMM eorrelation matficei. This section introduces a termi^ 
nology of generalised covariance component models appropriate for icaje free 
^alyiii. Discuiiion li reitricted to itructural characteristics of OCAi parame- 
ter eitimates are always obtained with the LISREL (Joreskog Sorbom, 1986) 
program^ 



3.1 Covariance component itructures 

In the original formulation by Bock & B^gmann (1966), covariance component 
analysis describes the facet-structured observed variables as linear functions 
of underlying latent vmates. The set of measures ihowi the latent itructura 
decomposition 



(14) 



The matrix H shows one variate for general latent vwiationj t variates for the 
traits, and m variates for the methods. The itructural coefficient matrix is 
assumed to be fully known and, for an MTMM matrix with 3 traits and 3 
methods, takes the form 



r 



A ^ 



0 
1 
0 
0 

1 

0 
0 

1 

0 

traits 



0 ^ 

0 
0 
0 
0 
0 

1 
1 

1 ; 

methods 



0 
0 
0 
1 
1 
1 
0 



0 0 
0 0 



(15) 



The correspondence to EquaMon 8 is apparant. The expectation of the sample 
covariance matrix Sa = -^yX' (I— All')X ii 



(16) 



with # being the covmance matrbe of the latent variates and 0 the (typically 
diagonal) eovariance matrix of unique and error components. Error variances 
(the diagonal entries in 0) may or may not be reitricted to homoscedasticity. 
Bock c§£ Bargmann originally assumed that # is diagonaL This aisumption is 
unnecessarily strict for applied purpoies, however, and Wiley et al. (1973) and 
Joreskog (1978) have extended the model to include correlated latent structures. 

GraybiU (1961, pp. 228--229) has shown that, since A in Equation 16 is not of 
fuUrankj not all parameters in # and 0 can be estimated. However, eitimation 
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of the eisenfcial variance components can be attained via reparameterization. 
To thii effect, two matrices K and L with 



A ^ K ' Ii, (17) 

are chosenj so that K is a matrix of m + 1 — 1 orthonormd column contrasts in 
A and L is a matrix of m -f i — 1 orfchogonal row contrasts of A with 



L^(K'K)^iK'A^K'A. 



(18) 



X may then be a expreiied in terms of a reduced n X (i + m— l) latent structure 
matrix H* ^ S ' L' as 



X, 



(tixmt) 



(19) 

(20) 



Instead of #, the covajiance matrix of the latent components, now #* = 
the cov^iance matrix of orthogonal transforms of the original latent 
components, is astimated. The matrix has two fewer rows and columns 
than but, since the omitted parameters could not be estimated in the first 
place, no information is effectively lost. Inteipretation must be based on the 
transformed parameters in which correspond to the three groups of vmatesi 

one variate for the general level of covariation, 

t — 1 variates deecribing differences in covariation due to traitSj and 

m — 1 variates expressing differences due to methods. 

The reparameteriEation traniforms Equation 16 into 

^ f (S^) = K#*K^ + 0, (21) 

Interpretation of the covariance components in #* muit reflect the particular 
choice of contrasts in K in addition to the empirical covariance structure. In 
the case of a 3 traits by 3 methods measurement design, for inetancej K may 
be chosen as 



K = 



/ 1/3 


vf/3 




0 


\/2/3 0 '\ 


1/3 


-i/vts 


1/ 




%/2/3 0 


1/3 


-i/vH - 


'1/ 


%/6 


s/2/3 0 


1/3 


%/2/3 




0 


-l/s/l§ l/yf 


1/3 




1/ 


s/6 


-1/i/lI l/yf 


1/3 


-i/vH - 


■1/- 


s/6 


-i/\/T8 1/vf 


1/3 






0 


-l/y/lS -1/VS 


1/3 


-i/vls 


i/- 




-l/%/l8 -1/Vt 


I 1/3 


-l/v/l8 - 






-l/VlS -l/^^ ; 


g 


traits 






methode 



(22) 
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Then the first trait contrast will reflect individual differences between trait 1 
ajid the other twoj the lecond one expreises difference variation between traits 
2 and 3, When the variaDce due to these centrasti ii laro, the original traits are 
indifl-inguiihable (i.e., perfectly eon-elated). Itaiti can only be diitinguished 
when the contrast vmates show non-iero vm^ca. Method variates would be 
interpreted coweepondingly bm differentia responiei to asieiament methods. 

When iubstantive conslderationi permitj the contraat matrix K ihould be 
simplified at one of the following levels* One, column contrasts for general, 
trait, and method components may be chosen m blockwise orthogonal. This 
allows testing independence between the trait and method differences and the 
general vaxiate. Two, if aU columns in K are orthogonal, correlations derived 
from #* may be directly interpretedj but the variance estimates will stiU be 
functionally dependent on the scale of the contrasts. Finally, when all contrasts 
are orthonormal (i.e., orthogon^ and normalised to unit le- gth), all parameters 
in ^e estimated on the same scale Mid latent v^ian ;es can be compared 
relative to each other. Qrthonomal contracts are advantageous when the rela- 
tive contribution of trait or method facets is assessed. All data an^yses in this 
paper are based on orthonormal contra t matrices. 

Several types of covariance component models, defined by restrictions of the 
matrix #*, should be distinguished. 

Fully correlated The observed cov^iance matrix can be expressed as a 
compound of trait and method variance components: 





(gymm.) 




a* 






Mr 


MM 



(23) 



There is some justification for trait concepts, but general, trait, and 
method variates are coirelatad, 

fcdependant-c«>mmon-variationi The first row and column show zero en= 
tries in the off-diagonal elements^ 





(symmO 


0 






0 







(24) 



'Erait and method variation is independent of the general variate, but trait 
and method contrasts may be intercorrelated. 

-^^ttetb.od indepeddencei ^i'ait contrasts are independent of method c jti = 
trasts (#*^ = 0), but the general factor may covary with either. 





(symm.) 










0 


■MM . 



(2F) 
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Block-diagonal IVait confcrasfcs are unco^alafced with mathod contraits 
and J in addition i the general v^iate is mdependent of both trait and 
method contrastp, 





(iymm,) 


= 0 






V 0 


0 





the empirical meaauremant structure has a block-diagonal covariance 
component form^ then the following three conclusioni ^e legitimate: (a) 
pattenis of individual differences in traits do not predict individual dif- 
ferencQi in response to methods; (b) differential reiponsa to method does 
not predict an individual's average level on aU measurasj and (c) trait 
contrasts do not predict the individuaPs relative itanding on the general 
variate. 

magonnl All (reparametari^ed) variates ^e uncomlatad* DiagonaUty is 
postulated by design only in the case of 2" measuramant designs. When 
trait or method facets contain mora than two alementSj diagonality wiU 
partly depend on the particular choice of contrasts. In theie casesj con- 
trast selection must be guided by subitMitiva theory. Diagonality implies 
that the researcher has, in substantive tarmSj found a most parsimonious 
account of the observed covmanca structure. This transcends the question 
whether trait and mathod diffarences are independent. 



3p2 Covariance component analysia with unknown scale 
factors 

Fixed-scale CCA in the form of Equation 21 calls for known or hypothesized 
scales of the latent vaxiatas over the entire set of measures or, aJternativelyj na- 
cassitatas specific assumptions about the uniqueness components in the diagonal 
of 0. Knowledge of the scale of maasurementj however, is often not available 
and, just as frequent ly^ is of secondly interest in analytie behavioral research. 
For instance, scale information in correlation analysis is lost entirely due to 
standardisation. Furthermore, multitrait-multimathod analyses are frequently 
conducted when fixed-scda assumptions across different traits Mid methods are 
not meaningful on conceptual grounds. Stand^disatien of observed variables 
in these cases imposes an arbitrary ceiling on the variance of the observed vari- 
ables and, in order to obtain any kind of intarpretable estimates for #* in the 
linear model framework, the relative true score scales have to be eitimated. 
Wiley, Schmidt & Bramble (1973) propose a class of scale-free ganeralizations 
of Equation 21 by introducing a diagonal matrix G of scaling constants: 

- GK#*K'G + 0. (27) 

G wiU absorb scale differences among the observed measures and should be 
interpreted accordingly. Wiley et aL (p. 317) state that 
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The major utaifcy of . [G] is for deaUnf with thoiQ iituations in 
which fcha observed variables are measured in different metrics. For 
such cases the introduction of . . , [G] whose elements do not have 
to be related to the variances of the variables allows for optimal 
rescaling* 

3«2«1 Mod©! identiflcation 

One element in G and #* must be set to a positive value to fix the scale of 
the estimates. This is because the Gramian product in Equation 27 is fenerally 
not identified due to a scale trade-off between and G: multiplication of 
wth a posifciva const ant a is fully compeniated for when G is iimultaneouily 
divided by y/a. This trivial underidantification has no coniequence for the 
substantive interpretation of the parameter estimates* Only the relative size of 
the component variajice and covariance estimates in #* is required to reconstruct 
the latent correlation structure of the measures. Furthermorej estimates of the 
scaling constants in G ^e only meajiingful in conjunction with the estimate of 

The underidantification is removed by a single non-^ero constraintj all ex- 
emplary analyses wiU use the identity 

- 1, (28) 

restricting the vmance of the general variate to unity. Variance estimates for 
the tr^t and method contrasts have to be evaluated relative to the variance of 
the general vwiata. 

An additional and more complicated identification problem arisei with fully 
correlated and traii-method independence CCA. It turns out that correlations 
with the general vmate; i.e., the elements of ff*^ and ^e unidentified in 
the scaJe-free model when tha measurement design is sm^l. Figure 1 illustrates 
this identification problem with a simple two-dimensional case. In both parts 
of the Figure J the length of the gener^ v^iata ^ has bean fixed to unity ^ 
^ 1.0). Case a describes the latent measurement structure as diagonal or 
block-diagonal and with unequal scale factors (Ai = 1.0 and — 0.3). G^a h 
describes the identical measurement structure using a fully correlated version of 
with ff*^ corresponding to a coirelation of coi(36.9°) = 0.80 and equal sized 
scale factors (Ai — ^ 0,72). Many other equivalent solutions exist the 
estimation equations are undefcerminad. 

The identification problem is practically independent of the size of the mea- 
surement design. It certainly remains when all but one contrast v^iates are of 
length zero. When the design is larger than 2 traits by 2 methods, trait-method 
independence CGA may be numerically identifiadj but some asymptotic corre- 
lations among tht estimatas usudly exceed 0.95. Thanj the precision of the 
parajneter estimation will not be acceptable. 

2 3 
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Figure 1: Two equivalent eovariance component reprf^entations 




In ihort, scale-free diagonalj block-diagonal, and mdependent--eominon-vari^ 
ation CCA models B^e genarally identified, while fche scale-free vemoni of the 
fully correlated and trdt-method independence models are not. Theea reiults 
contradiet iome of the claims by Wiley et al. (1973). Correlafcioni of the com- 
mon variata can be evduated if the scale faetorj are known. Alternatively, 
icale factors can be detarmined whan strict asinmptions about the eon-elation 
itructure of the common vaiiafce are imposed. 

3,a Empirical application of scale-free CCA 

This section applies scale-free block- diagonal and independent- common'VariaiiQn 
OCA to the 23 empirical MTMM matrices. Both models ^e direct generaliza« 
tions of congeneric trait-only factor ajialysis. Model performance is evaluated 
in terms of idantification, convergence, admissibility, and fit. Definitions for the 
first three criteria are identical to thosa for factor analysis in Section 2.1. 

Admissibility of CCA solutions implias that 0 and #* are both non-negative 
definite. Admasibility shall also denote that no latent variate will account for 
more than the total vmanca of the measured variables^ 

0 < gi\ki^^^\a%j < . (29) 

For correlation matricai, the upper bound becomes unity. 

Models using the other three covariance component itructures; i.e,, strictly 
diagonal, trait-method independence, and fully coirelatad CCA ^e not evalu- 
ated here for different reasons. The diagonal submodel requires theory-guided, 
not simply design=guided, selection of contrasts and transcends the scope of this 
paper. The other two submodels allow for correlation between contrasts and 
the general variate and ^e ridden with identification problevns. 

Table S summarises the icala=free block=diagonal covariance component anal- 
yses of the 23 datasets. All solutions are identified ajid converged j but not aU 
are admissible. Solutions for 13 of the 23 datasets violate the admissibility con- 
ditions. Five of ^;he 10 admissibla solutions have good fit ["^Popularity &nd Ex- 
pansivenesi^(Borgatta)**j ^Smoking and Capital Punishmant (Jaccard)^, ^Job 
Behavior (Dickinson U Tice)^, ^Job Performance (Lawlar)^, and Attitudes 
to the Church (Oatrom)"]. With an additional dat^at ["Assessment (Kelly U 
Fiske)^] the fit is m^ginal. Four of these solutions show a significant fit increase 
comp^ed to the '^ongeneric trait-only factor solution, while one [^Smoking and 
Capital Punishment (Jaccajd)**] was already well fit by tha congeneric two-fcrait 
factor model, hi one case ["Job Behavior (Dickinson & Tice)^] tha trait-only 
factor model had not converged to an admissibla solution, while admissibil» 
ity was achieved for the "Three Attitudes (Flamerj Smpla 1)^ data using the 
trait-only modelj but not in the c^e of block- diagonal CCA. 

The indepan dent-common variation modal is a generalization of block-diag- 
onal strueturas. Allowing for non-zero covariancas between trait and method 




Table Si Scale-free covmance component analysis of the datasets, 
block^diagona lm odal. ^ 

Admissible 

Dataset _ Identified Conyerged Soltttio n df N 

BitjUigenea - - _ 

and Effort no 0.0 0 166 

(MayQ) 

BitaUigenea " - 

& Alertness 17^2 S 750 

(Thorndika) 

Popurarity & ~ " — - 

Expansiveiiesi 14^6 13 jgS 

(Borgatta) 

Smoking " -^-"^ 

and CP 9^9 13 35 

_( Jacc^d) 
Leadership 

Study (Sum- no 81.4 13 196 

meri et aJ.) 

Authority ~ ~ - = 

(Burwen ^ no 16.8 S 57 

Campbell) 

Drives in _ _ _ _ _ 

^^^^ no 3.5 S SO 

(Anderson) 

Involvement ~ ~ 

Components 50^4 2I 96 

(Arora) 

Job Behavior ~ ~ ~ 

(Dickinson 16.3 21 149 

^ Tice) 

Three Afcti- " ~~~ 

tudes (Flamer^ no 19.3 21 105 

Sample 1) 

Three Atti'^ ~~~ ~ ' 

tudes (Flamerj no 35.9 21 105 

Sample 2) 

Stress - - — .^^^ 

Measures no 46.4 21 80 

(Karst U Mos t) 

Job ~~ ~~ - - 

Performance 29,2 21 113 
(Lawler) 



Table 5 — ^Continued 





Adnuesible 








Dataiat 


iaentxtied Converged Solution 




df 


N 












(bnephardj 




131.3 


21 


487 


Gontracsp^ 










tives (Kofc^ 
handapani) 


no 


104.5 


45 


100 


^ttitudas to 










tha Church 
(Ostrom) 




58.4 


45 


189 












Raports 


no 


224.2 


45 


190 


(Stacy et aL) 










Clinical 










Clerkships 
(Boodoo) 


no 


143.1 


77 


136 


Personality 










Traits 

(Kalley Krey) 


no 


15.2 


13 


311 


Deiirability 










(Jackson ^ 




848.8 


164 


480 


Singer) 










Interaction 










Process Vars. 


no 


211.9 


77 


125 


(Borgatta) 










Guilford- 










Martin Fact. 
(CarroU) 


no 


182.7 


77 


110 


Assessment 










(Kelly & 
Fkke) 




104.7 


77 


124 



21 

26 



ERIC 



Table 6: Scsle-frte covaxiance component analyiia of the dat^ets, indepen- 
dent-common-variafcion model 



Datasat 


Admiseible 
Idantifiad Converged Solution 


df 


N 


InteUigenca 
and Effort 
(Mayo) 


not idantiflad (the design ii too EmaU) 






Intelligence 
4§£ Alertneii 
(Thorndike) 


no 4.4 


3 


750 


Popularity ^ 

Bxpajiiivaneii 

(Borgatta) 


no 10.7 


10 


125 


Smoking 
and CP 
f Jaccard) 


8.6 


10 


35 


Leadariliip 
Study (Summarij 
et al) 


no 67.0 


10 


196 


Authority 
(Bujwen & 
Campbell) 


noi the first parameter (gn) is not identified 






Drivae in 
Rats 

(Andarion) 


no 3.4 


3 


SO 


Involvamant 
Componants 
(Arora) 


41.3 


17 


96 


Job 

Bahavior 

(Dickinion & Tica) 


14.0 


17 


149 


Three Attitudas 
(Flamari 
Sample 1) 


no 17.2 


17 


105 


Three Afctifcudei 
(Flamer, 
Sample 2) 


no 29.9 


17 


105 


S trass 
Maasurei 
(Karit Most) 


no 43.1 


17 


80 





Table 6~Cont%nued 





AdmisBible 








Dataset 


Identified Converged Solution 




df 


N 


Job 










Parformanca 




22.3 


17 


113 


(Lawler) 










Moral 










Dilemma 
(Shepherd) 




94.8 


17 


487 


Goatracej^ 










tives (Kot^ 
handapani) 


no 


87.3 


39 


100 


Attitudes to 










the Church 


no 


62.6 


39 


189 


(Oifcrom) 










Drug Uie 










Iteports 
(Stacy efc aL) 


no 


207.7 


39 


190 


Clinical 










Clerkihips 
(Boodoo) 


no 


138.9 


69 


136 


Pei^onality 










IVaits 

(Kelley U Krey) 


no 


7.7 


10 




Desirability 
(Jackson & 




798.5 


142 


480 


Singer) 










Interaction 










Proceis Vara. 


no 


1S7.9 


69 


125 


(Borgatta) 










Guilford- 










M^tin Fact. 


no 


153.2 


69 


110 


(CanroU) 










Assessment 










(Kelly k 
Fiike) 




87.4 


69 


124 



29 

28 



ERIC 



contrasts, (i — 1) K (m - 1) additional parameteri are estimated from the data. 
Resulti are summarised in Table 6. Solutions for two dataaets are not identi- 
fied^ wltli the "Intelligence and Effort (Mayo)^ data, the 2x2 measurement 
design is simply too smallj whUa the "Authority (Burwen & Campbell)** correla- 
tion matrbc app^ently presents m empiricai identification problem. Estimation 
procedures converged to an admissible solution in only 7 cases. Fit improve- 
ment is small in the four cases where the block-diagonal model already ap- 
proximates the empirical cowelation matrix [^Smoking and Capital Punishment 
(Jaceard)'*, -Job Behavior (Dickinson & Tice)^, "Job Performance (Lawler)"]^ 
and ^Assessment (Kelly & Piske)^. With three other dataaets, for which the 
block-diagonal structure did not provide an acceptable accountj fit improve- 
ment by the in dependent-common- variation model is substMitial but not large 
enough. For the "Involvement Components (Arora)^ (Diff-G^ = 9.1, df — 4), 
the "Moral Dilemma (Shepherd)^ (Diff-G^ = 36.5^ df ^ 4), and the "Desirabil- 
ity (Jackson ^ Singer)^ data (Diff-G^ = 50.3, df ^ 12), neither block-diagonal 
nor independent-common-variation covariance component structures yield close 
descriptions of the empirical coirelation matrices. 

The independent-common- variation model provides acceptable descriptions 
for four of the 23 empirical coirelation matrices. These matrices are, how- 
ever, already well accounted for by the more restricted block-diagonal covari- 
ance component modeL The incremental utility of the independent-common- 
v^iation model over the block-diagonal model therefore cannot be clearly af- 
firmed. This lack of significant improvement is quite likely a function of the 
particular dataiets used in this study and does not indicate any model defi- 
ciency. For the time being, the model may or may not be endorsed, pending 
some less ambiguous evidence becoming available. 

3,4 Interpretation of CCA iolutions 

In addition to finding the correct component structure, substantive interpreta- 
tion of the parameter eitimates is a necesswy p^t of the data analysis. Unfor- 
tunately, in my experience, it is quite a difficult enterprise to explain covariance 
component estimates (in 4*) to social scientists and even to some trained statis- 
ticians. Training in contrast techniques appem to be latkinf * Prior knowledge 
on these matters does, howeveri aid in the underst^ding of this section; Bock 
(1975) md Finn (1974) provide useful terminolofy on these matters. 

As a typical example for MTMM analysis. Table 7 displays a contrast matrix 
K for the Kelly and Piske asiessment data previously discussed in Table 4 in the 
factor analytic context. K contains seven contrasts, one for the general variate, 
four for trait vM-iation, wid two for method variation. The matrix is columnwise 
orthonormal; i.e., the contrasts are uncorrelated and have unit length. The 
component values for the general variate (in the first column) are standard 
and should not be modified. Yetj since the data analysis was only concerned 
with determiiimg the overall covmance component structure, and no further 



Tabli 1\ Orthonormal contrast matrix K for the Kelly Is Fiake asiessment data. 



iVl6EAOa 


IraiC 














— ? ri 


Staff 


Aeaertiye 


.25820 


.51640 


0.0 


0,0 


0.0 


.36515 


0.0 1 


Ratmgi 


Cheerful 


.25820 


-.12910 


.5 


0.0 


0.0 


.36515 


0.0 






.25820 


-.12910 


-.16667 


.47141 


0.0 


.36515 


0.0 




Poisa 


.25820 


-.12910 


-.16667 


-.23570 


.40825 


.36515 


0,0 




Xntareiti 


.25820 


-.12910 


-.16667 


-.23570 


-.40825 


.36515 


0.0 


Team- 


Assart ive 


.25S20 


.51640 


0.0 


0.0 


0.0 


-.18257 


,31623 


IP ite 


Cheerful 


.25820 


-.12910 


.5 


0.0 


0,0 


-.18257 


,31623 


Hatmgi 


Serioui 


.25820 


-.12910 


-.16667 


.47141 


0.0 


-.18257 


.31623 




Poiie 


.25820 


-.12910 


-.16667 


-.23570 


,40825 


-.18257 


,31623 




Interesti 


,25820 


-.12910 


-.16667 


-.23570 


-.40825 


-,18257 


.31623 


Self 


Assertive 


,25820 


.51640 


0.0 


0.0 


0.0 


-.18257 


-.31623 


Ratingi 


Cheerful 


.25820 


-.12910 


.5 


0.0 


0.0 


-,18257 


-.31623 




Sarioui 


.25820 


-.12910 


-.16667 


.47141 


0.0 


-.18257 


-.31623 




Poise 


.25820 


-.12910 


-.16667 


-.23570 


.40825 


-.18257 


-.31623 




Interasts 


.25820 


-.12910 


-.16667 


-.23570 


-.40825 


-.18257 


-.31623 



confirmatory substantiva hypofcheiei ware employed ^ tha Halmert contrasts in 
the trait and method blocks were chosen arbitrarily. Tha first trait contrast 
detenninas the latent variata as the difference bttwaan Assertiveneas and 
the remaining four traits, S^^ describes the difference betwaan OheerfulnBsa and 
the avaraga of Sertouamss^ UnshakahU Poise, and Broad Interasts, 6^^, finally, 
contraits Unshakable Poise with Broad Interests. Coiraipondinglyi the method 
contrast is dafined to absorb the difference between Staff Ratings and the 
average of Teammate and Self Eatings^ while compares Teammate Ratings 
against Self Ratings, 

Fit of the block-diagonal CCA model is marginally significant (with G- ^ 
104.7 ajid df » 77) and can be considered satisfactoryj given that the corre- 
lations were computed from ratmg ic^es. Estimates for model Equation 27 
are dkplayed in Tables 8 and 9. The uniqueness coefficients in Table 8 

have the same interpretation as their factor ualytic equiv^ent. Self ratings of 
Assertiveness^ Cheerfulness, and Senousness have uniqueness components al- 
most twice m large as the coirasponding ratings obtained from teammates and 
staff members, indicating that the self ratingi are less reliable, reflect different 
insights and standards, and/or are mediated by additional constructs like the 
person's degree of confidence. Teammate and Staff ratings differ most notica- 
bly for Unshakable Poise, the unique component being twice as l^ge for the 
teammate data. 

The scale factor estimates (G)« in Table 8 reflect differences in "true score^ 
variance of the obierved measures. Measures associated with larger scale fac- 




Table Bi Estimafced scale factors G and uniqueness coafficienti 







Scale 


Un iqueness 


Method 


Tredt 


Factors (G)i* 


Coeffs. 


Staff 


Assertive 


1.628 


.250 


Ratings 


Cheerful 


1.650 


.330 




Serious 


.774 


,639 




Poise 


2.253 


.425 




Interests 


1.836 


.533 


Team- 


Aisertive 


1.616 


.255 


mate 


Cheerful 


1.468 


.478 


Ratings 


Serious 


.956 


.404 




poise 


1.126 


.837 




Laterests 


1.907 


.418 


Self 


Assertive 


.925 


.714 


Ratings 


Cheerful 


.987 


.709 




Serious 


.557 


.811 




Poise 


1.725 


.569 




Interests 


1.975 


.367 



tors discriminate on a relatively larger scakj above and beyond the ays tern a tic 
variance due to the covariance component struct ure K#*K-. Such an inter- 
pretation is eo^^ect for MTMM covariance matrices. When generalized CCA is 
bajed on MTMM correlation matrices, instead, scale factor estimates will also 
be dependent on the error variMice of the original (unitandardized) measures. 
Then, the substantive interpretation of (G)*; wiU be less direct. In either case, 
the diagonal of (G) contains the estimated scale factors needed to optimally 
rescale the original variables aa Y ^ XG^^, transforming Equation 8 to 

By « G-iE^G-i (30) 
- G^^(GK#*K^G + 0)G^i (31) 
« K#*K^ + G-^0G^i (32) 

All three ratings of Broad Interests are found to show scale factors of com- 
parable magnifcuda. AsaertivenBss , Cheerfulness ^ and Seriousness have simito 
scale factors for staff and teammate rating methods, while self rating factors 
are substantially smaller. Self ratings of these variables are not comparable to 
^ufcings made by othera. For the Kelly & Fiske assessment data, scale factors 
^ifl ^Uiiiqueness coefficients reflect different aspects of the same phenomenon. 

^jitifes In 4* aje h^der to interpret than scale factors and uniqueness coef*- 
fi^etnti^ G«ven the contrast definitions in Table 7, the astimates can be viewed 




Table 9: Eitimated Covajiajice Component Matrix 4*. 













Sr. 






1,0* 












6f, 

Sn 


0.0* 

o.o* 


,670 
.387 


.793 




(symmetric) 


Su 


0.0* 


^.639 


-.821 


1.284 






Sr. 


0.0* 


-.087 


,107 


.020 


.16€ 


} 




0.0* 


0.0 


0.0 


0.0 


0.0 


,126 


s^. 


0.0* 


0.0 


0,0 


0.0 


0.0 


,100 .360 



aa relative variance eomponenti of the raipe stive latent variatea^ The firit com^ 
ponent (#fi) is fixed at unity and definie the icale of ^ other eitimatai in ft* 
and G. The remaining diagonal element i of #* contain the relative ^wianca due 
to the trait and method eontrasti. Bubitantive interpretation of the variance 
components is only meanmgful when the coiresponding contri^t has itself any 
Sttbstwtive significaiice. For instance, ^an be undent ood as the variance 

of the difference between staff ratings %^i^ui teanMnate and lelf ratings. On the 
other handj trait contrasti had been chosen wbitrarilyj so that the estimates 
in #*^ have no direct substantive interpratation. While it is not inconeeivabla 
to attempt a direct interpretation of the estimates in , the result of such an 
attempt would appe^ contrived and nearly incomprehensible. 

Similar interpretative problems appear for the off-diagonal elements in #*; 
Le., for the covariance components that reflect the association ajnong the con- 
trasts variates. The moderately positive coif elation between the two method 
variates can, for instance, be interpreted to relate all fating $ by others: People 
with higher average values in staff ratings than in combined mate and self ratings 
also tend to be rated higher by the teammates than they rate themielves. Here 
again, considering the present dataset, direct interpretation of trait covajiance 
components is not easily communicated. 

When contrasts we M'bitrarily selected, as in the present c^e, some purely 
exploratory transformation of the solution may be required for substajitive in= 
terpretation. In the case of block-diagonal CO A, and only than, can several 
transfonnations from the tool box of the multivariate literature be reasonably 
employed, such as disperaion component comp^isons, mnoniml decomposition 
of the covariance component blocks, and blockwise vartmam rotation, 

3«4.1 Disperiion eomponenta 

The determinant of a covariance matrix is frequently regarded as a scalar mea^ 
sure of the generalized multivariate variance (Green ic OairoU, 1976; Kendall & 
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Stuart, 1968; Wilks, 1932). It is wall known (cf., Searla, 1982, p. 258) that the 
datamiinant of a block-diagonal matrix equals the product of the determinants 
of the blocks. When the covariance component solution is block-diagonal with 







(symm.) ^ 


0 










0 




datenninMit |#*| is f 


acilitate 












- 1 







(33) 



(34) 



In the case of generalized CCA, with the scale conefcraint of — 1, the deter- 
minant simplifies further to 



The dispersion of the whole covmMice componant matrix #* tharafore equals 
the product of trait and method dbpenioni. With the KeUy U Fiske asiessment 
data, both dispersion components are imallj with 0*0122 and 0.0354, respec- 
tivelyj mdicating that moit of the syitematie variance of the optimally-scaled 
ratings is due to the general variate 6g, Method differencei account for ilightly 
more variation in the Aiiessment data than trait differences. 



3*4^2 Canonical decomposition of covariance component blocks 

^terpretation of covariajice component estimatei is greatly facilitated whan 
the diagonal blocks and can be trwiiformed into a diagonal structure. 
Choleski faefcorization Eigenvalue da compos it ion are wa Unknown traditional 
methods for this purpose* 

Under Choleski factorisation, a nonnegative definite symmatric matrix A is 
decomposed into the product of a lower triangul^ matrix S and its transpose: 
A - SS'j with S'B diagonaL Computational procedures are described in many 
texts, for instance, Anderion (1984), Bode (1975), Finn (1974), and Maindonald 
(1984). 

The results of Choleski factoriiation depend on the order of calculation. If 
A is of order g k g, Uiere will be gl numerically different Choleski factors S 
with the equivalent product SB' " A. Order dapendenee of S may be put to 
im advantage when the contrast variates can ba entered by importance or, in 
reverse order j by dubiosity. Then, af^, tha squared first diagonal entry in S, 
containi the relative variance due to tha most important contrast, jf^ is the 
partial variance of tha second most important contrast, adjusted for effecte of 
the first one, a^^ the p^tial contribution of the third contrast, adjustad for the 
first two, ^d so on* These values may be evaluated in step-down fashion as 
successive partiaj contributions. 




Unfortunately, there affe many instances when an importance ranking of the 
latent v^iatei is not meMiingful on iubitantive groundi; the present analysis of 
the Kelly & Flake cessment data being one of them. When the vajiates cannot 
be ranked beforeh^d, values in the diagonal of B m w^bitr^ and cannot be 
interpreted by themselves. Yetj even in this case, Gholeski factorization will 
furnMh a canonical matrix decompoiition in the forni A = SiS'j effectively 
reducing the covmmcB matrices ^% and to orthononn^ ^wiates. The 
derived soHtion can then be further rotated to aid interpretation (see below). 

Eigenvalue decomposition is another well-known method to describe a matrix 
m terms of a canonical structure. Eigenv^ues and the coiresponding (non- 
zero) Eigenvectoi^ of a symmetric matrix A are defined as the roots of 

Aq£ ^ qiXg. (36) 

Solutions can be obtained by v«ious numerical methods, many of which are im= 
plemented m such maint^ed software Ubrmes as IMSL (B18L, 1977), MAT- 
CAL (Bock & Repp, 1974), and the NAG Ubr^ (NAG, Ltd., Oxford, U^K.). 

Eigenvectors associated with different Eigenvaluei of the lame symmetric 
matrix are orthogonal. All Eigenvectors of A may be scaled to unit-langth and 
assembled in the columns of the matrix Q, so that Q^Q ^ L By collecting the 
associated Eigenvalues in the same order in the diagonal matrbc D^, Equation 36 
can be written more compactly as 

AQ = QDj,. (37) 

This further implies the canonical decomposition 

A^QD^Q^ (33) 

It has become customM^y to base the mterpretation of Eigenanalysis on the 
weighted principal components P * QDp rather than on the normaU^ed com^ 
ponents Q. 

The size of the EigenvJues in Dj^ reflects the ^wiance of the respective 
Bigencomponenti of A- qJAq^ s A^. It is well known that the largest Eigenvalue 
is the size of the largest v^iance component in A, the second largest Eigenvalue 
the variance of the Iw^gest component that is orthogonal to the first, etc. (e.g., 
Anderson, 1984). The size of the Eigenvalue becomes a useful indicator for 
empirical impOTtance of the principal componenti of a covmance matrix. SmaU 
variance componenti Me likely redundant, 

_^ Since ^bitrary choice of trait or method contrasts affects the estimate of 
I computation of principal components must be based on the entire Gramian 
product K#*K', liL the block-diagonal CCA model, this covariance structure 
can be additively partitioned into gener^, trait, Bxid method components asi 

/ gg^ i (sym mQ 
K#*K' ^ (K^|K.|K^) -Q 




(39) 



- k^^'k; + k,#;,k; + k^#;^k;, (4o) 
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Table 10: Unrotated component loadmgi P of the Kally ic Fiaka data 



Method 


Trait 




Pti 


P«3 


Pu 


Pu 


Pmi 




Staff 


Aiiertive 


MB 


-.331 


-.233 


-.121 


.020 


-.079 


-.103 


Ratings 


Cheerful 


.258 


-.312 


.229 


-.053 


-.075 


«.079 


-.103 




Serious 


.258 


.733 


-.006 


-.063 


-.018 


-.079 


-.103 




Poise 


.258 


-.045 


.130 


.051 


.124 


-.079 


-.103 




Intereiti 


.258 


-.042 


-.120 


.190 


-.047 


-.079 


-.103 


Team- 


Aisertive 


.258 


-.331 


-.233 


-.121 


.020 


-.147 


.084 


mate 


Cheerful 


.258 


-.312 


.229 


-.053 


-.075 


-.147 


.084 


Ratingi 


Serioui 


.258 


.733 


-.006 


-.063 


-.018 


-.147 


.084 




Poise 


.258 


-.046 


.130 


.051 


AM 


-.147 


.084 




Interesti 


.258 


-.042 


-.120 


.190 


-.047 


-.147 


.084 


Self 


Aisertive 


.258 


-.331 


-.233 


-.121 


.020 


.227 


.018 


Ratings 


Cheerful 


.258 


-.312 


.229 


-.053 


-.075 


.227 


.018 




S^ious 


.258 


.733 


-.006 


-.063 


-.018 


.227 


.018 




Poiae 


.258 


-.045 


.130 


.051 


.124 


.227 


.018 




biter esti 


.258 


-.042 


-.120 


.190 


-.047 


.227 


.018 


Variance 




1.0 


2.246 


.415 


.182 


.072 


.239 


.054 



Separate Eigenitructurei ihould be computed for the trait component TLf^tr^l 
Mid the method component K^^J^Kj^. 

Table 10 shows the unrotated principal component i of the Kelly Is Fiake 
assesiment data, computed from the block diagonal CCA solution. AU ieven 
vmates are now unconrelatad ^d have unit vajiancei thm columni of compo- 
nent loadingi are weighted contrasti, sorted within blocks with respect to the 
explained variance* The first trait component j p^j has more thw twice the 
variance of the fener^ variata. It is clearly defined as a contrast between Seri- 
ousness on one hand and the two varfables A3$ert%vene3^ and Cheerfulness on 
the other. Ssriousnsss has the largest trait component— this variable is most 
distinct from the Sg^ the general variatej and, consaquently, frnm most of the 
remaming traits in the study. AssertiveneMs md Cheerfulne&s are further re- 
moved from Seriousns&s than Poise and Broad Interests^ The remaming trait 
components ^e relatively minor: pt^ through pt^ reflect some diffarancas be- 
tween the four trait domains other than Seriousness^ Finallyj the two method 
components indicate that most method variance is due to the difference between 
self ratings and ratings by others. 

3*4,3 Blockwlse VAREMCAX rotation 

Varimax rotated canonic id components (Kaiser, 1958) comprise the same infor- 
mation as theu" unrotated count erp arts and as the covariance component matrix 
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Table 11: Rotated component loadmgs P of the Kelly & Fiske data 



Method 


Trait 


^0 




F«3 


Pu 


Pu 






Staff 


Asiertive 


.258 


-.407 


-.078 


-.030 


.076 


-.014 


-.129 


Ratmgi 


Cheerful 


.268 


-.084 


-.374 


.064 


-.086 


-.014 


-.129 




Serioui 


.258 


.500 


.481 


.176 


.171 


-.014 


-.129 




Poise 


.258 


.018 


-.043 


.019 


-.185 


-.014 


-.129 




Intereiti 


.25^ 


-.025 


.016 


-.231 


.022 


-.014 


-.129 


Taani- 


Aiiertive 


.258 


-.407 


-.078 


--030 


.076 


-.170 


-.004 


mate 


Cheerful 


.258 


-.084 


-.374 


.064 


-.086 


-.170 


-.004 


Ratmgi 


Serious 


.258 


.500 


.481 


.176 


.171 


-.170 


-.004 




Poke 


.258 


.018 


-.043 


.019 


-.185 


-.170 


-.004 




htereiti 


.258 


-.025 


.016 


-.231 


.022 


-.170 


-.004 


Self 


Assertive 


.258 


-.407 


-.078 


-.030 


.076 


.184 


.134 


Ratmge 


Cheerful 


.258 


-.064 


-.374 


.064 


-.086 


.134 


.134 




Serious 


.258 


.500 


.481 


.176 


.171 


.184 


.134 




Poise 


.258 


.018 


-.043 


.019 


-.185 


.184 


.134 




Uteres ti 


.258 


-.025 


.016 


-.231 


.022 


.184 


.134 



exeept that the rotation leads to a iimple structure solution. Many students 
are familiax with iimple itructure solutioni in factor analyeis, from where it is 
a minor step to the intfi^retation of a simple itructure derived from covariance 
componenti. 

To demonstrate such a iimple structure j the trait and me: hod blocks in Ta- 
ble 10 were iubjected to separate vmmax rotationi. The reiult is shown in 
Table 11. Evidence in the trdt block now cle^-ly shows that the four iimple 
contrasts between Ssrioumesa and each of the other traits are uncorrelated. The 
effecjbs iizei of the rotated components corroborate earlier findings: Assertive^ 
nesM and Cheerfulness B^m most diipM^ate from SeriousneMa^ while UnshakabU 
Fotae and Broad Mteresis are located iomewhat closer. The rotated method 
components also appear as independent iimple contrasti: The first component 
showi a difference between self ratmgs md Uammate ratings, the second in- 
die at ei that self ratings and staff ratings vary in different direetions. Both 
rotated method component a how about equal siie. 

3.5 Discusiion 

This paper studies the performance of two closes of multivariate Hne^^ model 
itructures for the multitrait-multimethod matrix: eonfirmatory factor analysis 
Biid generalized covariance componeni analysis. Notable submodels with appli- 
cationi to multitrait-multimethod ^alysis ^e identified in each class. Trait- 
only Mid independent trait^method decomposition models af e ielected from fac- 
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tor analysiSi block' diagonal and ind^pmndanUcommon^variation represent the 
covM'ianea eamponent appro The four models are partially nee ted and two 
itrandi of hierarchical modal testmg may be pnritied. The two factor wialytic 
modeli can be directly compared to each ©ther and the statiitical iignlficance 
due to added method fact Of a can be teifced comparing the two model likelihoodg. 
The other line of nested hier^chieal model testing aUowa comparisons between 
the trait-only factor modal and the two generalised CCA models. 

Performance of all four model typei is evaluated with 23 empirical MTMM 
matrices using the criteria of identification j convergence, admiisibilitys and 
model fit. 

With these dataj the trait-only factor model was generally found to be iden- 
tified and convergedj yet the solutions were often inadmlsaiblaj and model fit 
was typically veiy poor. Howevarj this is a posit iva result compared to the 
trait-inethod factor modelj which never even converged to an admissible so- 
lution. Formal evidence is provided ihowing that the trait-method factor is 
rotationaUy (and conceptually) underdetannined, Thw m bad news, because 
the trait-method decomposition model has been extensively promoted in the 
literature and its deficiencies ara not yet widely known. 

Two types of identification problems ara found with generalized covariance 
component models. The first is trivial and easily removed* because the scale 
factors ara estfimatad, the scale of the vaiiance componants is lost, making it 
necessary to fix a single variance component or one scaling constant at a non-zero 
value. Only the relative size of covariance component estimates is maaningfuL 
The second identification problem is mora severer scale-free generalization of 
covariance component an^ysis finds its limitations when the common wiate is 
allowed to correlate with the contrast components. Iil the presence of any kind 
of empirical sampling en-or; i.e., in bII empirical applications, estimation of these 
parameter groups gives very unreliable results. Estimation of these correlations 
is only possible when the latent scale of the measured v^iables is assumed to be 
known (and vice versa). This new finding qualifies some of the very optimistic 
statements by Wiley et al. (1973). 

Block- diagon^ and independant-common-vmation CCA converged to ad- 
missible solutions in about half the cases j with acceptable model fit for 5 or 6 
of the 23 dataseti. The fact that just a moderate number ©f MTMM matrices 
could be successfully modeled m a favorable result, considering that for several of 
these datasets multitrait-multimethod validation would have bean questionable 
on substantive p^ounds already. A good statistical model must be falsifiable on 
empirical grounds to be of any practical use. 

Despite its impressive estimation properties, covariance component analysis 
hm not gained nea^ as much popularity among researchers as the competing 
factor analytic model. One of the major reasons for this development may 
be that the interpretation of covariance componenti is difficult and unpopular. 
To facilitate inte^retationj primal estimates of co^riance components can 
be transformed into canonical variates and rotated into simple structure or 
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into other orientations that may be heuriitically halpfuL Worked examplei we 
provided. 

Covariance component ^alysis provides a fundamental vehicle for the asseeg- 
ment of trait validity. Evaluation of validity should be based on a compreheniiva 
model of the p^amefcers that underlie an. obierved congelation etructure, rather 
than the individual sampk correlation components themielves. In reference to 
the treatment by GampbeU U Fiske (1959) it is observed that aonvergeni va- 
lidity m reflected by dia appearing method covaiianee itructures; i.e., » 0, 
while discriminant validity m establi£hed when the deteoninant of the trait co- 
variance component matrix ^% m l^ge. The covaxiance structure approach h^ 
the interesting implication that traits can be validated in obiermtional itudiee 
only insofar m they differ from other traits in the same study. 
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Deicriptio^3i of DatMets 

Model perforniance ii evduaSad ^^Eth 23 empirical coirelation matrices pub- 
Ji^shed in fcha psychological and iocidld^gical literature, Nina matrices ^e taken 
fra-'om the original article by Campbell Fiske (1959) ^ the other 14 datasets were 
^contributed in VMiouspape^ wrifcteji ^^ince, A iynoptic characterisation of each 
IVfflCTMM matrix iiprovidad in Table L _2. 

The size of mgaiuriment deiign the datasets varies from 2 traifee by 2 

?n-^ath©di to 4 traitiby 5 methodi and traits by 3 methodi. Sample sizes range 
t^stween 35 and 7E Informatioii on ^meani and vM^iancee was notably absent 
fr<^iOm all reportSj riEicfcing the tfaditiiiMnal neglect of the scale of measurement 
Jii ^ much of piycholopal reiearch' 

The trait domdni ^e variously tfo^^iteivad m abiHtiei; locial dispositions or 
behavior, attitudes or attitude c^^mponentSs drives j and social desirability 
j^^*-dginenti. Attracdvineis of diffaranSb methods as study objects hm changed 
markably over the yam: while G^^pbeU Is Fiske comp^ed mostly effects of 
^elLJf ratingSi ratingi by others^ obj^^ctive measures, later studies concern the 
#l!^acts of different quiition formats oi^-^ of different panels engaging in political 
pr*^^ferance judgments* 

Two studies show unorthodox metlfe^od concepts* AU six measures in the "In- 
etlDligence and Alartneis (Thorndika)^ ^data are paper-and-pencil aasessments of 
^bi^ility with ^^InteUigfinea^ and ^'Manta^ Alertness" labeled as traits and "^Mem- 
c?r^r^j ^Compreheniion"j and ^VbcabuMary" labeled as methods. The ^CUnical 
(JU^wkships (Boodooj* data assess g^»e^^^ dispositions in **Pediatrics^ , "Internal 
H^^dicine'^j and "Surgary** as traits^ t^Mad on the performance in such method 
^o^-inains as "Skills^ Troblem solviji^^j etc. In these two cases, both facets 
the meaaurement diiign are of sub-r-^tantive interest and the **trait^ versus 
'^R^^athod*' distincfcion becomes arbitral— y. Boodo© (1985) did, in fact, label her 
f^c^ats the exact oppoiita way. One ahot^ld keep in mind that multivmate meth- 
o^^& just as easily accommodate croii-cl^BSsifications among several trait facets as 
th^^y can handle mgasurement designs c^f traits by methods. Hierarchical model 
t^STsting is aided from i substantive pci^ait of view when the facets clearly differ 
i*i t xalevancej but eitimition methods a^aid fit statistics remMs unaffected even 
wH*^an all facets araaqudly importaiit* 

One of the corrilation matricaSj t^i^a ^Moral Dilemma (Shepherd)^ data^ 
pTQoved not to be poiltiva definite and, consequently, not Gramian. This may 
hav^j^a been due to a typgiatting a^^or or some pairwise deletion of missing data 
r^Jt»iaining unreportad. For the preiant ^mn^yses, the originally published matrix 
W^^^ smoothed subtracting the negatlvi i xoots from the correlation structure and 
a^^^ing a ridge of smaU variance conipo : jients to the diagonal. 
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Table 12i Description of datasete 



Nain6j Trait- Size Domini 



Conception 


Tbraiti Methods N 




Traits 




Methodi 


Data Source 


Intalligence 


2 2 166 


1 


Intelli- 


a 


Peer 


Campbell and 


and Effort 






gence 




Ratings 


Piike (1959) 


(Mayo) 




2 


Effort 


b 


Objective 


Abilities 










Me as urea 




Intelligence 


2 3 750 


1 


Intelli- 


a 


Memory 


Campbell and 


and 






gence 


b 


Oompre'^ 


Fiske (1959) 


Alertness 




2 


Mental 




hension 


(Thorndike) 






Alertness 


c 


Vbcabu^ 




AbilitieE 










lary 




Popularity 


2 4 125 


1 


Popul^ity 






Ci ATT! ft ^1 1 STirt 


and 










Self 


Fiike f 19591 


Exp anil ve- 










Rating 


ness 




2 


Ep^ansiva^ 


b 


Rating by 




(Borgatta) 






neii 




Others 




Social 








c 


Observed 




Disposi- 










Group In^ 




tions 










teraction 












d 


Observed 














Role 














Playing 




Smoking 


2 4 35 


1 


Cigarette 


a 


Semantic 


Jaccard, 


and 






Smoking 




DiffereU' 


Weber, and 


Capital 




2 


Capita 




tial 


Lundmaxk (1975) 


Punishment 






Punish^ 


b 


Likert 


( Jaccard, 






ment 




Scaling 




et aL) 








c 


Thurstone 




Attitudea 










Scaling 












d 


Guilford 














Scaling 




Leade^.'ship 


2 4 290 


1 


Community 




Panels of 


Summers, 


(Summers, 






Leadership 


a 


IT School 


Sailer, and 


et al.) 




2 


Educa^ 




Laadei^ 


Wiley (1970) 


Social 






tional 


b 


20 Organi- 


Diaposi^ 






Leadership 




sation 




tions 










Heads 












c 


19 Popu- 














lar Judges 












d 


196 Heads 














of Houie'* 














holds 
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Table 12— Continued 



Name; I^ait^ Sfae Domaini 



Coneaption 


TVaits Methods 


N" 




Ttaiti 




Methods 


Data Source 


Afcfcitudti 


3 2 


57 




Attitudes 






Campbell and 


to Authority 






1 


To father 


a 


Interview 


Fiiks (1959) 


(Burwen and 






2 


To boss 


b 


Check 


Campbell) 






3 


To peer 




List 




Attitudes 














Drives in 


3 2 


50 


1 


Hunger 


a 


Obstruc- 


Campbell and 


Rati 






2 


Thirit 




tion Box 


Fiske (1959) 


(Anderson) 






3 


Sex 


b 


Activity 




Drives 














Involvement 


3 3 


96 




Involvement 




Rating Scale 


Arora (1982) 


Gomponenti 






1 


Situa» 


a 


Stapel 


(Arora) 








tional 


(j 


Likert 




Attitudes 






2 


Enduring 


c 


Semantic 










3 


Responie 




Different 

tid 




Job 


3 3 


149 


1 


Getting 


a 


Peer 


Dickinson & 


Behavior 








along with 




Nominations 


Tice (1973) 


(Dickinion 








others 


b 


Peer Check= 


o£ X ice j 






2 


Dedication 




list 




Bocial 






3 


Ability to 




Ratings 




Dispogi^ 








appty 


c 


Self Check- 




tions 








learning 




list 
















Ratings 




Three 


3 3 


105 




Attitude 


a 


Likert 


Flamer (1983) 


Attitudes 








Tow^ds 




Scales 


(Flamer, 






1 


Disci- 


b 


Thurstone 




Sample l) 








pline of 




Scales 




Attitudes 








children 


c 


Semantic 










2 


Mathennatics 




Differ- 










3 


The law 




ential 




Three 


3 3 


105 




Attitude 


a 


Likert 


Flamer (1983)"" 


Attitudes 








Towards 




Scales 


(Flamer, 






1 


Disci- 


b 


Thurstone 




Sample 2) 








pline of 




Scales 




Attitudes 








children 


c 


Semantic 










2 


Mathematics 




Differ- 










3 


The law 




ential 
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Tabla 12~Cont%nuBd 



Namaj Trait- 


Size 




Oomi 








Conception 


IVaits Methods N 




Traiti 




Methods 


^Data Source 


Stress 


3 3 80 




Straai 




Oeneral 


Kajst and 


Meaaurai 




1 






Self K^f^ 


iVlOSy I -L^ < - J 


(Karat and 






^ ^ b Y 




ings 


Most) 




2 


i-J m IJl K 


□ 


Anchored 




Arouaal 






Perform- 




Self 




States 






anca 




Ratings 








3 


Post Per- 


c 


Finger 










£ormance 




Sweat 














Print 




Job Per- 


3 3 113 


1 


Quality 








formance 






of job 


a 


^ 1 1 ^ ^T"! rtT*3 


(Lawler) 






parform= 


b 


Peers 




Soeial 






anca 


c 


Self 








2 


Ability to 








tions 






perform 














job 












3 


Effort put 














forth on 














the job 








Moral 


3 3 487 




Morality 




Three 


Shepherd (1977) 


Dilemma 




1 


Negative 




differeiit 


(Shapherd) 




3 


Positive 




test forms 




Baliefs and 




3 


Achievement 








Attitudes 






of 








Attitudes to 


3 4 100 




Attituda 






Kothandapani 


Contracap- 






Components 


a 


Thurstone 


(1971) 


tives 




1 


Affective 




Scaling 
Likert 




(Kofchanda'^ 




2 


Behavioral 


b 




pani) 




3 


Cognitive 




Scaling 




Attitude 








c 


Guttman 




Components 










Scaling 












d 


GuiUbrd 














Sealing 
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Tabla 12~ContinuBd 



Namaj Traifc» She Domains 



Conception 


Ttaite 


Methods 


N 




Itaits 




Methods 


Data Source 


Attitudes to 
the Church 


3 


4 


189 




Attitude 
Componantg 


a 


Thurstona 


Ostrom (1069) 


(Ostrom) 








1 


Affactiva 




Scaling 




Attitude 








2 


Behavioral 


b 


Likert 




C omp onentp 








3 


Cognitive 


e 

d 


Scaling 

Guttman 

Scaling 

Guilford 

Scaling 




Drug Use 


3 


4 


190 


1 


Alcohol 


a 


Self 


Stacy et aL 


Reports 








2 


Marijuana 




Rating 


(1985) 


(Stacy 








3 


Nicotine 


b 


Self 


at aL) 

Social 

Behavior 












c 
d 


Intake 

Report 

Peer 

Rating 

Peer 

Intake 

Report 




CUnical 
Clerkshipi 


3 


5 


136 


1 


Pediatrics 


a 


Teacher Ratings 
SkiUs 


Boodoo 
(1985) 


(Boodoo) 








2 


Internal 


b 


Problem 


Social 










Medicine 




Solving 




DispQii-^ 
tionM 








3 


Surgery 


c 

d 

e 


Relation- 
ihips 

Knowledge 
Attitude 




parson aUty 


4 


2 


311 




Social Itaiti 






Campbell and 


^aits 
(KaUy and 








1 
2 


Courtesy 
Honesty 


a 


Peer 
Rating 


Fiike {1K9) 


Krey) 
Social 








3 
4 


Poiaa 
School 


b 


Aisocia- 
tion 




Disposi^ 










Drives 




Test 





tions 



Table 12~Coniinued 



Name, 11fait= 


Size 




_ ^ — 


oj^ns 






Conceptioii 


Traits Methods N 




Traits 




lVX€uXlQQ9 


Oat a Source 








Judgments on 






Jackson (1975) 








€ia on ail by 












J.ralE9 


a 


Oesirable 








J. 


Fein ill inity 




in bell 




itv Tpq^ifs 






Anxiety 


D 


JJesiraDie 








O 


Soiiiat Ic 




in Othera 










wOinpiaiiiiia 


c 


w nay 








A 
% 


oociaiiy 




Others 




on, person- 










Find Desir= 
















a Die 










A 


jrequeJlcy 














of Occur- 














rence 












e 


TT 

Aarmiui" 














ness 










bocial Seliavior 






Oampbell and 


Procasi 




i 


Shows Sal- 




JVee 




Variables 












(Sorgatta) 




2 


Vi?lV€a i^UgB 


D 






Social 










flaying 








3 


^5ives 


C 


Pr oj e c t ive 














±€Sb 








4 


Gives Orien^ 














tation 












5 


Shows Dii- 














afreement 








Guilford- 


5 3 110 


1 


B 


a 


Inventory 


Campbell and 


Martin 




2 


T 


b 


Self 


Fiske (1959) 


Factors 




3 


D 




Rating 


(Omoll) 




4 


C 


c 


■ Peer 




Personality 




6 






Rating 




Diiposi^ 














tions 














Clinieal 


5 3 124 


1 


Assertive 


a 


Staff 


Campbell and 


Aiseisment 




2 


Cheerful 




Rating 


Fiike (1959) 


(KaUy and 




3 


Serious 


b 


Teammate 


Fiske) 




4 


Unshakable 




Rating 




Social 






Poise 


c 


Self 




Itaits 




B 


Broad 




Rating 





^terests 



44 



References 

Alfchauserj R.P. (1974) Inferring validity from tha multitraifc»multimethod ma- 
trix: ajiother ^sessment. In H.L. Oostner [Ed.] Sociological Methodology 
1973-1974, San BVanciseo^ Jossey-Basi 

Alfchauieri R.P. h Hebarlein, T.A. (1970) Validity and the multitrait=multi- 
method matrix. In E.F. Borgatta [Ed.] Sociological MBthodology 1970, 
San JVmciicoi Joisey-Baii 

Althausar, Hebarlem, T.A., U Scottj R.A. (1971) A causal aasaiimant of 

validity! the augmented multitrait-multimethod matrix. In H,M. Blalock 
[Ed.] Causal models in the social sciences, Chicago: Aldine 

Anderion, T.W. (1984) An iniroduction to multivariaie statisticah analysis 2nd. 
Ed.). New Yorki Wiley 

Anderion^ T.W. Rubin, H. (1956) Statistical inference in factor analysis. Pro^ 
ceedings of the Third Berkeley Symposium of Mathematical Statiaiics and 
Probability, 5,111^150 

Aroraj R. (1962) V^dati of an S-O-R model for iituationj enduringj ^d re= 
sponse componanti ©f involvament. Journal of Marketing Research^ 19j 
605^516 

Bealsj R.J Krantij D.H., k Tvarskyj A. (1968) Foundations of multidimensional 
scaling. Psychological Review, T5, 127-142 

Bock, R.D. (i960) Oomponanti of variance analyiis as a structural and discrim- 
mal analysis of psycholofical tests. The British Journal of Statistical Psy- 
chology ^ 13(2), 151^163 

Bock, R.D^ (197S) Multivariate Btatistical methods in behavioral research. New 
York: McGraw=Hill 

Bock, R.D. U Bargmannj R.E. (1966) Analyiis of cov^iMice itructures. Psy- 

chometrika, 31(4), 507-^534 
Bock, R.D., Dicken, C, & Van Pelt, J. (1969) Methodologies impUcationi ©f 

content" acquiesce nee carrelati©n in the MMPI. Psychological Bulletin, 

Tl(2), 127^139 

Boodoop G.M. (1985) The itructure of clinical evaluatien forms across three 
clerkships. Paper presented at the American Educaiional Research As- 
sociation Meeting^ Chicago^ April 1985, 

Browne, M. W. (1984a) Asymptotically distribution-free methods for the analysis 
of covariMice structures. British Journal of Mathematical and Statisticai 
Psychology, 37, 1-21 

Browne, M.W. (1984b) The decomposition of multitrait-multimathod matrices. 
British Journal of Mathematical and Statistical Psychology ^ 37, 62-83 

Campbell, D.T. (1969) Definitionid multiple operationaliam. et aL, 2, 14-17 

Campbell, D.T. & Piska, D.W. (1959) Convergent and diicriminant validation 
by the multifcrait-multimathod matrix. Psychological Bulletin^ 56, 81-105 

Dickinson, T.L. k. Tice, T.B. (1973) A multitrait-multimethod analysis of scales 
developed by retranslation. Organisational Behavior and Human Perfor- 
ms 



46 



mance, 9, 421-438 

ijrnn, J.E. (1973) A nofce on a suffieiency condition for uniqueneia of a restricted 

factor matrix. PsychomBirika, S8j 141-143 
Finn, J.D. (1974) A general model for multivariate data analysis. New York.! 

Holt, Rlnahart & Winiton 
Fiakgj D.T, (1982) Convergent-diaciiniinant validation in maasurement and re- 

sawch stratagies. New Direciions for Methodology of Social and Behavioral 

Science, W, 77-92 

Flamerj S. (1983) Aiieiimint of the mulfeitrait-multimathQd matrix validity of 
Likert icalei via conflrinatoty factor analysis. Multivariaie Behavioral 
Research, IS, 275-308 

Graybillj F.A. (1961) An introduction to Linear Statistical Modeh, VoL 1 New 
York: McGraw-Hill 

Jacc^d, J., Weber, X, & Lundma^-kj J. (1975) A multitrait^multimethod anal- 
ysii of four assassment procedures. Journal of Emperimenial Social Fsy- 
chology, 11, 149^154 

Jackson, D.N. (197S) Mulfcimefchod factor walyiis: a reformulation. Multivari- 
ate Behavioral Research i 10, 259-275 

Jennrich, R.I (1978) Rotafcionai equivalanca of factor loading matricei with spec- 
ified valuai. Psychometrika, 43, 421-426 

Joreskog, KG. (1966) Taitinf a simple stmctura hypothesii in factor analyiiB. 
Psychometrika, Slj 165^178 

Joreskogj K,G. (1971) Statiifcical aaialysis of seti of congeneric teiti. Psychome- 
trika, 36(2), 10^133 

Jdreekog, K.G. (1978) Structural analysia of covarianca and correlation matrices. 
Psychometrikaj 43 (4) , 443-^477 

Joreskog, K.G. U Sorbom, D. (1979) Advances in factor analysis and structural 
equation models. Gambridge, MA: Abt Books 

Joreskog, KG. U SSrbom, D. (1986) LISREL 6, Analysis of Linear Struc^ 
tural Relationships by the Method of Mamimum Likelihood. User's guide. 
Mooreiville, IN: Scientifie SoftwM^e 

Kalleberg, A.L. & Kluegel, J.R. (1975) Analysis of the muUitraifc-mulfcimathod 
matr^! some limitations and an alternative. Journal of Applied Psychol- 
ogy, 60(1), 1-9 

K^ife, T.O. is Mestj R. (19T3) A comparison of itress measures in an experi- 
ment^ analogue to public spewing. Journal of OonsulUng and Clinical 
Psychoiogy, 4(3), 342^348 

Kothandapandi, V. (1971) Validation of faeUng, belief, and intention to mt as 
three components of attitude and their eontribution to prediction of con- 
traceptive behavior. Journal of Personality and Social Psychology. 19(3} 
321-333 

Lawler, E.E., III (1967) The mulfcitrait-multimethod approach to measuring 
manaf erial job performance. Journal of Applied Psychology, 51(5), 369- 
381 



46 

47 



Lawley, D.N. {1940) The estimafcion of factor loadings by the mtthod of maxi- 
mum likelihood. Proceedinga of the Royal Statisitcal Society of Edinburgh 
60,64-^82 

Lawlay, D.N. Is Mwcwallj^ A,E. (1971) Fastor analysis as a staiistical method. 

2nd ad. London.' Bufctarworth 
Lords P.M. & Novick, M.R. (1968) Siatistiaal theoHes of mental test scores. 

Reading, MA: Addiion-Weilay 
Mamdonald, J.H. (1984) Statuiical Computation. Naw York: WHay 
Ostromi T.M. {1969) The ralationship betwean affective^ behavioral, ^d cogni- 

fcive components of attitude. Journal of Emperimental Social Psyeholoay 

5, 12^30 ^ y.. 

Schmitt, N. (1978) Patli analyiis of mnltitrait-multimethod matricas. Applied 

Psychological Measurement i 2(2), 157-173 
Searle, S.R. (1982) Matrim algebra useful for statistics, Naw Yorki Wllay 
Bhaphardi G. {1977) A critique and extension of Bronfenbranner's moral 

dilemma test. Internationa! Journal of Psychology^ 12(3)^ 207-217 
Staey^ A.W., Widaman, Hayij R., U DiMatfceo, M.R. (igs's) Validity of 

self-rapOTti of alcohol and other drug usei A multitrait-multimathod aa= 

sessmant. Journal of Personality and Sociai Psychology, 49{l)j 219-232 
Summers, SeUer, L.E. h Wilay, G. (1970) Community and educational 

laadarship. In E.F. Borgatta [Ed.], Sociological Methgdalogy. San SVan- 

ciaco^ Jossey^Basi 
Werti, C.E.J Joreskogj K.G. U Linn, R.L. (1972) 

A multitrait-multimetjiod model for studying growth. Educational and 

Psych 'Iznical Measurement, 32j 655-678 
mrts, C.B. j: an, R.L. (1970) Path malyaii. Piychological examples. Psy^ 

chologicci Bulletin^ W{3), 193-212 
Werts, C.E.J Linn, R.L.j Is Joraikog, K.G. (1971) Estimating the parametars of 

path models involving unmeasured variables. ^ H.M. Blalock [Ed.] 

Causal models in the social sciences, Chicago^ Aldine 
Wileyj D., Schmidt, W.H., U Bramble, W.J. {1973) Studies of a class of covm- 
mce structure models. Journal of the American Statistical Association 
68(342), 317-323 

Wothke. W. {1984) The estimation of trait and method components in multitrait- 
muitimethod measurement. University of ChicagOj Department of Behav- 
ioral Seiance^ Unpublahed Diseartation. 



47 

48 



